fixing the display on the AIX server solved the issue.

HCL Notes/Domino 8.5 Forum (includes Notes Traveler)

Subject:

we were getting the same error on trying from the Domino Console. On the server end, we found this error at server startup -

********************* Java VM problem *********************
Unable to start the Domino Console: Error reported by JVM:

No X11 DISPLAY variable was set, but this program performed an operation which requires it.
Skipping Console, continuing with Domino Server Controller.
***********************************************************

I used the following command "$export DISPLAY=0:0" to fix my display. And now am able to connect to the server through Domino Console.

Feedback response number WEBB8B4QBK created by ~Arnold Chujipyvitchgon on 11/11/2010